Methods for configuring tubing for interconnecting in-series multiple liquid-cooled cold plates

ABSTRACT

Methods of configuring a cooling subassembly for an electronics system are provided, that is, for establishing a coolant-carrying tube layout for interconnecting multiple liquid-cooled cold plates in series-fluid communication for cooling multiple heat-generating electronic components of an electronics system. The electronic components are to be plugged in fixed relation into a preconfigured motherboard, and the tube layout includes at least one rigid coolant-carrying tube. Simplified analysis is initially performed to evaluate stress on the rigid tube(s) and determine if loss of actuation load on the electronic components exceeds an acceptable threshold, and if so, at least one tube having high stress is identified and reconfigured. Thereafter, analysis is performed to determine whether, with available actuation load on the cooling subassembly, electrical connection loading between the electronic components and the supporting motherboard is above an acceptable minimum level. If so, the coolant-carrying tube layout is chosen as a final design.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application contains subject matter which is related to the subjectmatter of the following applications, each of which is assigned to thesame assignee as this application and each of which is herebyincorporated herein by reference in its entirety:

-   -   “Hybrid Cooling System and Method for a Multi-Component        Electronics System”, Campbell et al., Ser. No. 11/539,902, filed        Oct. 10, 2006;    -   “Conductive Heat Transport Cooling System and Method for a        Multi-Component Electronics System”, Campbell et al., Ser. No.        11/539,905, filed Oct. 10, 2006;    -   “Method of Assembling a Cooling System for a Multi-Component        Electronics System”, Campbell et al, Ser. No. 11/539,907, filed        Oct. 10, 2006;    -   “Liquid-Based Cooling System for Cooling a Multi-Component        Electronics System”, Campbell et al., Ser. No. 11/539,910, filed        Oct. 10, 2006;    -   “Heatsink Apparatus for Applying a Specified Compressive Force        to an Integrated Circuit Device”, Colbert et al, Ser. No.        11/460,334, filed Jul. 27, 2006; and    -   “Method and Apparatus for Mounting a Heat Sink in Thermal        Contact with an Electronic Component”, Colbert et al, Ser. No.        11/201,972, filed Aug. 11, 2005.

TECHNICAL FIELD

The present invention relates in general to cooling an electronicssystem, and more particularly, to a liquid-based cooling system forcooling a multi-component electronics system. Still more particularly,the present invention relates to methods for configuring acooling-carrying tube layout for interconnecting in-series multipleliquid-cooled cold plates of a liquid-based cooling system for cooling amulti-component electronics system.

BACKGROUND OF THE INVENTION

The power dissipation of integrated circuit chips, and the modulescontaining the chips, continues to increase in order to achieveincreases in processor performance. This trend poses a cooling challengeat both the module and system level. Increased air flow rates are neededto effectively cool high power modules and to limit the temperature ofair exhausted into the computer center.

In many large server applications, processors along with theirassociated electronics (e.g., memory, disk drives, power, etc.), arepackaged in removable drawer configurations stacked or aligned within arack or frame. In other cases, the electronics may be in fixed locationswithin the rack or frame. Typically, the components are cooled by airmoving in parallel air flow paths, usually front-to-back, impelled byone or more air moving devices (e.g., fans or blowers). In some cases itmay be possible to handle increased power dissipation within a singledrawer by providing greater air flow, for example, through the use of amore powerful air moving device or by increasing the rotational speed(i.e., RPMs) of an existing air moving device. However, this approach isbecoming unmanageable at the frame level in the context of a computerinstallation (e.g., data center).

The sensible heat load carried by the air exiting the frame willeventually exceed the ability of room air conditioning to effectivelyhandle the load. This is especially true for large installations of“server farms” or large banks of computer frames close together. In suchinstallations, not only will the room air conditioning be challenged,but the situation may also result in recirculation problems with somefraction of the “hot” air exiting one frame being drawn into the airinlet of the same or a nearby frame. Furthermore, while the acousticnoise level of a powerful (or higher RPM) air moving device in a singledrawer may be within acceptable acoustic limits, because of the numberof air moving devices in the frame, the total acoustic noise at theframe level may not be acceptable. In addition, the conventionalopenings in the frame for the entry and exit of air flow make itdifficult, if not impossible to provide effective acoustic treatment toreduce the acoustic noise level outside the frame. Finally, as operatingfrequencies continue to increase, electromagnetic cross talk betweentightly spaced computer frames is becoming a problem largely due to thepresence of the openings in the covers.

Accordingly, there is a significant need for enhanced cooling mechanismsfor electronic components, individually and at all levels of packaging,including for example, rack-mounted or blade-mounted electroniccomponents of various large computer systems today.

SUMMARY OF THE INVENTION

The need to cool current and future high heat load, high heat fluxelectronic components requires development of aggressive thermalmanagement techniques, such as liquid-based cooling systems and methodsof fabrication. The concepts disclosed herein address the need forenhanced liquid-based cooling systems for facilitating cooling of amulti-component electronics system, as well as the need for enhancedmethods of fabricating liquid-based cooling systems.

Briefly summarized, the present invention comprises in one aspect amethod which includes: obtaining a coolant-carrying tube layout forinterconnecting multiple liquid-cooled cold plates in series-fluidcommunication for cooling multiple heat-generating electronic componentsof an electronics system, the interconnected multiple liquid-cooled coldplates being a cooling subassembly, the multiple heat-generatingelectronic components to be plugged in fixed spaced relation intocorresponding sockets on a supporting motherboard, and thecoolant-carrying tube layout comprising at least one rigidcoolant-carrying tube; determining for the coolant-carrying tube layoutif stress in one or more rigid coolant-carrying tubes thereof exceeds apredetermined acceptable level, and determining if loss of actuationload on a heat-generating electronic component being cooled by thecooling subassembly exceeds an acceptable loss threshold, and if eitheris true, identifying at least one tube of the coolant-carrying tubelayout having high stress, the at least one tube comprising at least oneof a tube portion in torsion and a tube portion in bending when a forceis applied to at least one liquid-cooled cold plate of the coolingsubassembly, and reconfiguring the at least one tube having high stressto produce a reconfigured coolant-carrying tube layout and repeating thedetermining for the reconfigured coolant-carrying tube layout;performing analysis on the cooling subassembly and the multipleheat-generating electronic components of the electronics system once thecoolant-carrying tube layout experiences tube stress below the definedacceptable level and a loss of actuation load on the multipleheat-generating electronic components below the acceptable lossthreshold, the analysis including employing the available actuation loadon at least part of the cooling subassembly and evaluating electricalconnection loading between at least one heat-generating electroniccomponent being cooled thereby and the supporting motherboard into whichthe heat-generating electronic components electrically connect in fixedspaced relation; and saving the coolant-carrying tube layout as a finaldesign for interconnecting in series-fluid communication the multipleliquid-cooled cold plates of the cooling subassembly if the electricalconnection loading is above an acceptable minimum level.

In a further aspect, a method of configuring a cooling subassembly foran electronics system is provided. This method includes: obtaining acoolant-carrying tube layout for interconnecting multiple liquid-cooledcold plates in series-fluid communication for cooling multipleheat-generating electronic components of an electronics system, theinterconnected multiple liquid-cooled cold plates being a coolingsubassembly, the multiple heat-generating electronic components to beplugged in fixed spaced relation into corresponding sockets on apreconfigured supporting motherboard, and the coolant-carrying tubelayout comprising at least one rigid coolant-carrying tube; performingsimplified three-dimensional numerical analysis of stress and strain onthe cooling subassembly to determine for the coolant-carrying tubelayout stress on at least one coolant-carrying tube interconnecting atleast two liquid-cooled cold plates of the multiple liquid-cooled coldplates and loss of actuation load on at least one heat-generatingelectronic component of the multiple heat-generating electroniccomponents resulting from the coolant-carrying tube layout, thesimplified three-dimensional numerical analysis of stress and strainbeing performed without evaluating the electrical connection loading ofthe multiple heat-generating electronic components into the supportingmotherboard employing an available actuation load; determining for thecoolant-carrying layout if stress in the at least one coolant-carryingtube exceeds a predefined acceptable level and determining if loss ofactuation load on the at least one heat-generating electronic componentexceeds an acceptable loss threshold, and if either is true,reconfiguring the coolant-carrying tube layout until stress in the atleast one coolant-carrying tube interconnecting the at least twoliquid-cooled cold plates is below the predefined acceptable level andthe loss of actuation load on the at least one heat-generatingelectronic component is below the acceptable loss threshold; performingdetailed three-dimensional numerical analysis of stress and strain onthe cooling subassembly and the multiple heat-generating electroniccomponents of the electronics system once the coolant-carrying tubelayout experiences tube stress below the acceptable level and loss ofactuation load on the at least one heat-generating electronic componentbelow the acceptable loss threshold, the detailed three-dimensionalnumerical analysis of stress and strain including employing theavailable actuation load on the cooling subassembly to evaluateelectrical connection loading of the multiple heat-generating electroniccomponents into the sockets on the supporting motherboard; and savingthe coolant-carrying tube layout as a final design for interconnectingin series-fluid communication the multiple liquid-cooled cold plates ofthe cooling subassembly if the electrical connection loading isacceptable.

In a further aspect, a method of configuring a cooling subassembly foran electronics system is provided, which includes: ascertaining aconfiguration of an electronics system, the electronics systemcomprising a preconfigured supporting motherboard and a plurality ofheat-generating electronic components to be cooled, the plurality ofheat-generating electronic components to be cooled being electricallypluggable into corresponding sockets on the preconfigured motherboard;obtaining available actuation load for plugging the plurality ofheat-generating electronic components into the corresponding sockets ofthe motherboard and a required final load on each electrical connectionof the plurality of heat-generating electronic components into themotherboard; and providing a cooling subassembly employing theascertained configuration of the electronics system, the coolingsubassembly comprising a plurality of liquid-cooled cold plates.Providing of the cooling subassembly includes: laying out at least onecooling subassembly, the at least one cooling subassembly comprising atleast two liquid-cooled cold plates of the plurality of liquid-cooledcold plates coupled in series-fluid communication for cooling at leasttwo heat-generating electronic components of the plurality ofheat-generating electronic components, the at least two liquid-cooledcold plates being coupled in series-fluid communication employing atleast one rigid tube, wherein the laying out includes: selecting aconfiguration for the at least one rigid tube; and employing theavailable actuation load and required final load on each electricalconnection to determine if loss of actuation load on a heat-generatingelectronic component to be cooled by one of the at least twoliquid-cooled cold plates coupled in series-fluid communication is belowan acceptable threshold, and if so, employing the selectingconfiguration for the at least one rigid tube in the at least onecooling subassembly, otherwise, reconfiguring the at least one rigidtube to reduce reactionary force resulting therefrom on at least oneliquid-cooled cold plate of the at least two liquid-cooled cold platescoupled in series-fluid communication, the reconfiguring being repeateduntil loss of actuation load on the at least one liquid-cooled coldplate of the at least two liquid-cooled cold plates coupled inseries-fluid communication is below the acceptable threshold.

Further, additional features and advantages are realized through thetechniques of the present invention. Other embodiments and aspects ofthe invention are described in detail herein and are considered a partof the claimed invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts one embodiment of a conventional air-cooled electronicsframe with heat generating electronic components disposed in removableelectronics drawers;

FIG. 2 is a plan view of one embodiment of an electronics drawer layoutillustrating multiple electronic components to be cooled, in accordancewith an aspect of the present invention;

FIG. 3 is a partially exploded perspective view of an air-cooled heatsink apparatus, in accordance with an aspect of the present invention;

FIG. 4 is a partial perspective view of the air-cooled heat sinkapparatus of FIG. 3, in accordance with an aspect of the presentinvention;

FIG. 5 is a cross-sectional elevational view of an air-cooled heat sinkapparatus and electronic component assembly, taken (for example) alongline 5-5 of FIG. 3, in accordance with an aspect of the presentinvention;

FIG. 6 is a cross-sectional view of a portion of the air-cooled heatsink apparatus of FIGS. 3-5, illustrating a non-influencing fastenerarrangement in an actuated state, in accordance with an aspect of thepresent invention;

FIG. 6A is a cross-sectional view of the non-influencing fastener ofFIG. 6, shown in a non-actuated state, in accordance with an aspect ofthe present invention;

FIG. 7 is a flowchart of one embodiment of a method of mounting anair-cooled heat sink in thermal contact with one or more electroniccomponents, in accordance with an aspect of the present invention;

FIG. 8 is a plan view of the electronics drawer layout of FIG. 2illustrating one alternate embodiment of a cooling system for coolingcomponents of the electronics drawer, in accordance with an aspect ofthe present invention;

FIG. 9 depicts one detailed embodiment of a partially assembledelectronics drawer layout, wherein the electronics system includes eightheat generating electronic components to be actively cooled, each havinga respective liquid-cooled cold plate of a liquid-based cooling systemcoupled thereto, in accordance with an aspect of the present invention;

FIG. 10A depicts one embodiment of a liquid-cooled cold plate employedin the cooling system embodiment of FIG. 9, in accordance with an aspectof the present invention;

FIG. 10B depicts one embodiment of a liquid-coolant header subassemblyemployed in the cooling system embodiment of FIG. 9, in accordance withan aspect of the present invention;

FIG. 10C depicts multiple preconfigured coolant-carrying tubes employedin the cooling system embodiment of FIG. 9, in accordance with an aspectof the present invention;

FIG. 11 is a perspective view of one embodiment of a liquid-cooled coldplate and electronic component assembly, in accordance with an aspect ofthe present invention;

FIG. 12 is an exploded view of the liquid-cooled cold plate andelectronic component assembly of FIG. 11, in accordance with an aspectof the present invention;

FIG. 13 is a top plan view of one embodiment of a liquid-cooled coldplate (shown with the cover removed) for a cooling system, in accordancewith an aspect of the present invention;

FIG. 13A is a cross-sectional elevational view of the liquid-cooled coldplate of FIG. 13, taken along line 13A-13A, in accordance with an aspectof the present invention;

FIG. 14 is an isometric view of an alternate embodiment of aliquid-cooled cold plate for a cooling system, in accordance with anaspect of the present invention;

FIG. 14A is a partially exploded view of the liquid-cooled cold plate ofFIG. 14, in accordance with an aspect of the present invention;

FIG. 15 depicts an alternate detailed embodiment of a partiallyassembled electronics drawer layout, wherein the electronics systemincludes sixteen primary heat-generating electronic components to beactively cooled, each having a respective liquid-cooled cold plate of acooling system associated therewith, in accordance with an aspect of thepresent invention;

FIG. 16 is an isometric view of one embodiment of one coolingsubassembly of multiple cooling subassemblies employed in the coolingsystem for the electronics drawer layout of FIG. 15, in accordance withan aspect of the present invention;

FIG. 16A is an elevational view of the cooling subassembly of FIG. 16,showing coupling of two liquid-cooled cold plates thereof withrespective heat-generating electronic components to be cooled, inaccordance with an aspect of the present invention;

FIGS. 17A, 17B & 17C depict one embodiment of a method for designing acoolant-carrying tube layout for interconnecting multipleseries-connected liquid-cooled cold plates of a cooling subassembly, inaccordance with an aspect of the present invention;

FIG. 18 is an isometric view of two liquid-cooled cold plates to beseries-connected employing, in part, a rigid tube comprising a tubeportion in bending and a tube portion in torsion when a force is applied(as illustrated) during the processing of FIGS. 17A-17C, in accordancewith an aspect of the present invention; and

FIG. 19 is an exploded view of one partial stack embodiment of asupporting motherboard and a heat-generating electronic component towhich a liquid-cooled cold plate is to be coupled for facilitating thecooling thereof, in accordance with an aspect of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION

As used herein “electronics system” comprises any system containing oneor more heat generating components of a computer system or otherelectronics unit requiring cooling. The terms “electronics rack”,“electronics frame”, and “frame” are used interchangeably, and includeany housing, rack, compartment, blade chassis, etc., having heatgenerating components of a computer system or electronics system and maybe for example, a stand-alone computer processor having high, mid or lowend processing capability. In one embodiment, an electronics framecomprises multiple electronics drawers, each having multiple heatgenerating components disposed therein requiring cooling. “Electronicsdrawer” refers to any sub-housing, blade, book, drawer, node,compartment, etc., having multiple heat generating electronic componentsdisposed therein. Each electronics drawer of an electronics frame may bemovable or fixed relative to the electronics frame, with rack mountedelectronics drawers and blades of a blade center system being twoexamples of drawers of an electronics frame to be cooled.

“Electronic component” refers to any heat generating electroniccomponent of, for example, a computer system or other electronics unitrequiring cooling. By way of example, an electronic component maycomprise one or more integrated circuit dies and/or other electronicdevices to be cooled, including one or more processor dies, memory diesand memory support dies. As a further example, the electronic componentmay comprise one or more bare dies or one or more packaged dies disposedon a common carrier. As used herein, “primary heat generating component”refers to a primary heat generating electronic component within theelectronics system, while “secondary heat generating component” refersto an electronic component of the electronics system generating lessheat than the primary heat generating component to be cooled. “Primaryheat generating die” refers, for example, to a primary heat generatingdie or chip within a heat generating electronic component comprisingprimary and secondary heat generating dies (with a processor die beingone example). “Secondary heat generating die” refers to a die of amulti-die electronic component generating less heat than the primaryheat generating die thereof (with memory dies and memory support diesbeing examples of secondary dies to be cooled). As one example, a heatgenerating electronic component could comprise multiple primary heatgenerating bare dies and multiple secondary heat generating dies on acommon carrier. Further, unless otherwise specified herein, the term“liquid-cooled cold plate” refers to any conventional thermallyconductive structure having a plurality of channels or passagewaysformed therein for flowing of liquid coolant therethrough. In addition,“metallurgically bonded” refers generally herein to two components beingwelded, brazed or soldered together by any means.

As shown in FIG. 1, in rack-mounted configurations typical in the priorart, a plurality of air moving devices 111 (e.g., fans or blowers)provide forced air flow 115 needed to cool the electronic components 112within the electronics drawers 113 of the frame 100. Cool air is takenin through a louvered inlet cover 114 in the front of the frame andexhausted out a louvered outlet cover 116 in the back of the frame.

FIG. 2 illustrates one embodiment of a multi-component electronicsdrawer 213 having a component layout in accordance with an aspect of thepresent invention. Electronics drawer 213 includes one or more airmoving devices 211 (e.g., fans or blowers) which provide forced air flow215 across the multiple electronic components 212 within electronicsdrawer 213. Cool air is taken in through a front 231 of electronicsdrawer 213 and exhausted out a back 233 of the electronics drawer. Inthis embodiment, the multiple electronic components to be cooled 212include processor modules disposed below air-cooled heat sinks 220, aswell as (by way of example) multiple rows of memory support modules 232disposed between arrayed memory modules 230, such as air-cooled dualin-line memory module (DIMM) packages.

Electronic components are generally packaged using one or moreelectronic packages (i.e., modules) that include a module substrate towhich the device is electrically connected. In some cases, the moduleincludes a cap (i.e., a capped module) which seals the electronic devicewithin the module. In other cases, the module does not include a cap(i.e., is a bare die module).

Bare dies are generally preferred over capped modules from a thermalperformance perspective. In the case of a capped module, a heat sink istypically attached with a thermal interface between a bottom surface ofthe heat sink and a top surface of the cap, and another thermalinterface between a bottom surface of the cap and a top surface of theelectronic device. In the case of a bare die, a heat sink is typicallyattached with a thermal interface between a bottom surface of the heatsink and a top surface of the electronic device. Bare dies typicallyexhibit better thermal performance than capped modules because bare dieseliminate two sources of thermal resistance present in capped modules,i.e., the thermal resistance of the cap and the thermal resistance ofthe thermal interface between the cap and the electronic device.Accordingly, bare dies may be preferred for electronic components thathave high power dissipation.

Air-cooled heat sinks are attached to modules using a variety ofattachment mechanisms, such as clamps, screws and other hardware. Theattachment mechanism typically applies a force that maintains a thermalinterface gap, i.e., the thickness of the thermal interface extendingbetween the heat sink and the module. In the case of a capped module,the cap protects the electronic device from physical damage from theapplied force. In the case of a bare die, however, the applied force istransferred directly through the electronic device itself. Consequently,when bare dies are used, the attachment mechanism typically applies acompliant force to decrease stresses on the electronic component.

FIGS. 3-7 depict one enhanced mounting mechanism for holding anair-cooled heat sink in thermal contact with an electronic component.Generally stated, in this embodiment, the air-cooled heat sink apparatuscomprises a load frame having load springs and an open region thatexposes the electronic component. The load frame is mounted to a circuitboard on which the electronic component is mounted. The air-cooled heatsink is disposed on the load frame and has a main body in thermalcontact with the electronic component through a thermally conductivematerial. The air-cooled heat sink has load arms for engaging the loadsprings. A load plate extends between the load arms and has an actuationelement operative to displace the main body relative to the load plateand thereby resiliently deform the load springs and produce a load forcethat compresses the thermally conductive material to achieve a desiredthermal interface gap between the main body and the electroniccomponent. Non-influencing fasteners secure the air-cooled heat sink tothe load frame and maintain the desired thermal interface gap.

Referring to FIGS. 3-5, an air-cooled heat sink 220 is illustrated,which implements an improved process for mounting the heat sink onto theheat source, such as an electronic component. FIGS. 3-5 illustrate majorcomponents of an air-cooled heat sink apparatus 220 at a high level, andit should be understood that the number, type and configuration ofcomponents may vary depending upon the implementation. For example, theapparatus may contain a different number, type and configuration ofelectronic modules to be cooled.

As best shown in FIG. 3, air-cooled heat sink apparatus 220 includes twomain components, i.e., a load frame/spring assembly 302 and a heatsink/load arm assembly 304. Load frame/spring assembly 302 includes aload frame 306 and a pair of load springs 308. Load frame 306 ispreferably made of an alloy material chosen for its low creepproperties, such Zamak 8. Zamak 8, also known as ZA-8, is the trade namefor a zinc-based alloy, the primary components of which are zinc,aluminum, copper, and magnesium. Creep is the development over time ofadditional strains in a material. Creep depends on the magnitude of theapplied force and its duration, as well as the temperature and pressure.A material having high creep resistance is preferable in theconstruction of load frame 306 because creep deformation is to beavoided.

Load springs 308 are preferably made of an alloy material chosen for itshigh tensile strength properties, such as high strength music wire.Although two load springs 308 are shown in FIG. 3, those skilled in theart will appreciate that the present invention may be practiced with anynumber of load springs 308 (and load arms 310, which engage the loadsprings 308 as described below in the discussion of heat sink/load armassembly 304).

Load frame 306 is mounted on a printed circuit board 312. Referring toFIG. 5, fasteners such as screws 510 (two of which are denoted withdotted lines in FIG. 5) are used to attach load frame 306 to printedcircuit board 312. In one embodiment, four screws 510 (i.e., one neareach corner of load frame 306) pass through thru-holes in a backsidestiffener 512, an insulator 514 such as a polyimide, and printed circuitboard 312, and are received in threaded holes in load frame 306. Thisconfiguration advantageously allows access to screws 510 even when theheat sink/load arm assembly is attached to the load frame/springassembly.

Returning to FIG. 3, load frame 306 includes one or more open regions314 into which extends the heat source, e.g., an electronic component(not shown) mounted on printed circuit board 312. For example, a baredie may be mounted on printed circuit board 312 at the locationdesignated at the intersection of the cross-hairs shown in FIG. 3.

As shown in FIG. 3, load frame 306 includes four mounting projections316 to which the ends of load springs 308 are secured. Load frame 306also includes two downstop support projections 318 on which rest themid-sections of load springs 308.

One or more non-influencing fasteners 320 are used to secure heatsink/load arm assembly 304 to load frame/load arm assembly 302. By wayof example, four non-influencing fasteners 320 are mounted on load frame306. Each non-influencing fastener 320 is threaded into a boss 516 (FIG.5) of load frame 306. The non-influencing fasteners (NIFs) lock the heatsink in position without influencing the position of the heat sink.

Heat sink/load arm assembly 304 includes a heat sink 324 having a baseplate 326. Preferably, heat sink 324 is formed with fins, pins or othersimilar structures to increase the surface area of the heat sink andthereby enhance heat dissipation as air passes over the heat sink. It isalso possible for heat sink 324 to contain high performance structures,such as vapor chambers and/or heat pipes, to further enhance heattransfer. For example, heat sink 324 may contain one or more vaporchambers (not shown) charged with deionized water. Heat sink 324 may,for example, be formed of metal, such as copper or aluminum, or of otherthermally conductive material, such as graphite-based material.

As mentioned above, heat sink/load arm assembly 304 includes load arms310. Load arms 310 are hingedly attached to a U-channel load plate 328.Load arms 310 and U-channel load plate 328 may be made of stainlesssteel, for example, and be configured to provide minimal air flowimpedance across the fins of heat sink 324. For example, load arms 310have an open area through which air may flow. When heat sink/load armassembly 304 is attached to load frame/spring assembly 302, load arms310 engage load springs 308. This engagement is described in detailbelow with reference to FIGS. 4 & 5. In addition, when heat sink/loadarm assembly 304 is attached to load frame/spring assembly 302,non-influencing fasteners 320 are received in bore holes 330 in the heatsink's base plate 326. This non-influencing fastener arrangement isdescribed further below with reference to FIGS. 5-6A. To aid inalignment of heat sink/load arm assembly 304 with respect to loadframe/spring assembly 302, load frame 306 may include alignment pins332, which are received in corresponding alignment holes (not shown) inthe heat sink's base plate 326.

FIG. 4 is a perspective view of a heat transfer apparatus 220 withportions of heat sink 324 removed. FIG. 5 is a cross-sectional view ofheat transfer apparatus 220 engaging an electronic component assembly.As shown in FIGS. 4 & 5, an actuation mechanism applies a preload forceto heat sink 324 toward a semiconductor chip 502 (FIG. 5) to compress athermally conductive material 508 (FIG. 5) and achieve a desired thermalinterface gap between heat sink 324 and semiconductor chip 502. The maincomponents of the actuation mechanism include load frame 306, the loadframe's mounting projections 316, load springs 308, load arms 310, theload arms' hook portions 410, hinge pins 412, U-channel load plate 328,actuation screw 414, push plate 520, the push plate's guide pins 334,heat sink 324, and the heat sink's base plate 326. Referring to FIG. 3,load arms 310 each include a hook portion 410 that engages one of theload springs 308. Load arms 310 are hingedly attached to U-channel loadplate 328 by hinge pins 412. An actuation screw 414 is threaded throughU-channel load plate 328 to engage an underlying push plate 520 (FIG.5). Actuation screw 414 may be, for example, an M3 screw. Actuationscrew 414 is accessible for actuation from the top of U-channel loadplate 328. The distance between the U-channel plate and push plate 520is adjusted by turning actuation screw 414. This provides a controlledrate of loading. Those skilled in the art will recognize that otheractuation elements and techniques to provide a controlled rate ofloading are possible within the scope of the present invention, such ascamming, rocking and the like.

Still referring to FIG. 4, when the load frame/spring assembly and theheat sink/load arm assembly are brought together, hook portions 410 ofload arms 310 are engaged with load springs 308, and the actuationmechanism is actuated by turning actuation screw 414 in a direction toincrease the distance between U-channel load plate 328 and theunderlying push plate 520 (FIG. 5). Load springs 308 are deflected byactuation of the actuation mechanism. The geometric parameters of loadsprings 308, (i.e., the span, cross-section profile, and diameter) areoptimized for the allowable space within the application and therequired resulting load. Force is transmitted through the heat sink'sfins and base plate 326 onto the underlying semiconductor chip 502 (FIG.5). The force compresses a thermally conductive material 508 (FIG. 5)and achieves a desired thermal interface gap between heat sink's baseplate 326 and semiconductor chip 502.

Referring to FIG. 5, push plate 520 is affixed to heat sink 324. Forexample, push plate 520 may be soldered to heat sink 324 using, forexample, SAC 305 solder. Alternatively, push plate 520 may be affixed toheat sink 324 with a suitable adhesive, such as epoxy. Push plate 520may be made of stainless steel, for example. In one embodiment, pushplate 520 is affixed in a location directly above the heat source, withthe width of U-channel load plate 328 and push plate 520 substantiallycapturing the footprint of the heat source. This provides centroidalloading above the bare die, and thus provides substantially no edgestress on the die. As shown in FIG. 5, for example, push plate 520 isaffixed to multiple heat sink's fins lying above semiconductor chip 402.Although not shown in FIG. 5, additional modules residing on printedcircuit board 312 may be accommodated in open area 314 of load frame306. In such a case, push plate 520 may be affixed in a locationdirectly over the primary module, with the width of U-channel load plate328 and push plate 520 substantially capturing the footprint of theprimary module.

As shown in FIGS. 3 and 4, the push plate includes guide pins 334 thatextend through corresponding holes in U-channel load plate 328. Thepurpose of guide pins 334 is to align push plate 520 relative toU-channel load plate 328.

As shown in FIG. 5, in one embodiment, the heat generating electroniccomponent comprises one or more bare dies, including a semiconductorchip 502, a module substrate 504, and an electronic connector 506.However, those skilled in the art will appreciate that the presentinvention may be practiced using other types of heat sources such as oneor more capped modules and/or other electronic components. The bare dieshown in FIG. 5 is a single-chip module (SCM); however, those skilled inthe art will recognize that the spirit and scope of the presentinvention is not limited to SCMs. For example, those skilled in the artwill recognize that the present invention may be practiced using one ormore multi-chip modules (MCMs), or a combination of MCMs, SCMs and/orother electronic components/heat sources.

It is significant to note that the present invention allows a singleheat transfer apparatus to accommodate one or more modules havingdifferent footprints. Previous solutions required qualification ofindividual modules based on differences in footprint. The presentinvention overcomes this drawback.

The bare die is conventional. Semiconductor chip 502 is electricallyconnected to module substrate 504. Electronic connector 506, whichelectrically connects printed circuit board 312 to module substrate 504,may be a pin grid array (PGA), a ceramic column grid array (CCGA), aland grid array (LGA), or the like.

In some cases, electronic connector 506 may be susceptible to beingcrushed by the force applied by the actuation mechanism. This isproblematic not only from the perspective of possible damage toelectronic connector 506, but it also throws off the planarity of thestack (i.e., the module substrate 504 and semiconductor chip 502)relative to the heat sink's base plate which causes thermally conductivematerial 508 to form an uneven thermal interface gap. In such cases, oneor more crush protection elements 522 (denoted with a dotted line inFIG. 5) may be inserted along peripheral portions of module substrate504 between the bottom of module substrate 504 and the top of printedcircuit board 312. The crush protection elements 522 may be made of amaterial such as a polythermal plastic or the like.

Referring to FIG. 5, thermal interface 508 is made of a thermallyconductive material such as thermal gel, grease, paste, oil, or otherhigh thermal conductivity material. For example, thermal interface 508may be made of Shin-Etsu gel or grease with aluminum and/or zinc oxidespheres. Typically, thermal interface 508 is relatively thin so that itmay easily transfer heat away from semiconductor chip 502 towards theheat sink's base plate 326. The thickness of thermal interface 508extending between the bottom of the heat sink's base plate 326 and thetop surface of semiconductor chip 502 is referred to as the thermalinterface gap. As one example, the thermal interface gap is about 1.2mil.

Thermally conductive material 508 is dispensed on semiconductor chip 502prior to bringing the load frame/spring assembly and the heat sink/loadarm assembly together. To protect semiconductor 502 as these assembliesare initially brought together, a viscoelastic foam pad 530 may beinterposed between the lower surface of the heat sink's base plate 326and the upper surface of load frame 306.

Those skilled in the art will appreciate that the actuation mechanismshown in FIGS. 4 and 5 is exemplary, and that other actuation mechanismsmay be used to apply the preload force within the spirit and scope ofthe present invention. According to one embodiment of the presentinvention, once the preload force is applied to achieve the desiredthermal gap, irrespective of the actuation mechanism that applied thepreload force, one or more non-influencing fasteners are actuated tosecure the heat sink to the load frame and maintain the desired thermalgap.

As shown in FIG. 5, when the heat sink/load arm assembly is attached tothe load frame/spring assembly, non-influencing fasteners 320 arereceived in bore holes 330 in the heat sink's base plate 326. Once theactuation mechanism applies the preload force to achieve the desiredthermal interface gap, non-influencing fasteners 320 are actuated tosecure heat sink 324 to load frame 306 and maintain the desired thermalgap. One embodiment of a non-influencing fastener arrangement is shownin more detail in FIGS. 6 and 6A. FIG. 6 shows a non-influencingfastener 320 in an actuated state, while FIG. 6A shows non-influencingfastener 320 in a non-actuated state. Non-influencing fastener 320includes a screw 610 that is threaded into one of the bosses 516 of loadframe 306. Captivated on screw 610 are a split taper ring 620 and asolid taper ring 630. Preferably, the taper of split taper ring 620matches that of solid taper ring 630. Non-influencing fastener 320 isaccessible through bore hole 330 in the heat sink's base plate 326, andis actuated by turning screw 610 into the load frame's boss 516 so thatsplit taper ring 620 is expanded against the wall of bore hole 330 inthe heat sink's base plate 326. Non-influencing fasteners 320 areadvantageous because they can be actuated without significantly alteringthe thermal interface gap, as would be the case with a conventionalfastener.

FIG. 7 is a flow diagram of a method 700 for mounting a heat sink inthermal contact with an electronic component according to one embodimentof the present invention. Method 700 sets forth one order of steps. Itshould be understood, however, that the various steps may occur at anytime relative to one another. Initially, the bare die is soldered to theprinted circuit board 710. If a crush protection element is desired,then the crush protection element is inserted along peripheral portionsof the module substrate between the bottom of the module substrate andthe top of printed circuit board 720. The load frame is attached to theprinted circuit board 730. Thermally conductive material is dispensed onthe semiconductor chip 740. Next, the heat sink/load arm assembly isaligned and brought into contact with the load frame/spring assembly750. During step 750, the hook portion of each load arms is brought intoengagement with one of the load springs.

Method 700 continues with the application of a preload force using theactuation mechanism to set the thermal interface gap 760. During step760, the actuation screw is turned an appropriate amount to apply apreload force (e.g., 40 lbs) that provides the desired thermal interfacegap (e.g., 1.2 mil). In other words, some of the thermally conductivematerial is squeezed-out by the preload force to provide the desiredthermal gap. Once this point is reached, the assembly may optionally bethermally cured to set the thermal interface gap. Next, thenon-influencing fasteners are actuated to secure the heat sink to theload frame and maintain the desired thermal gap (step 770). Preferably,an appropriate torque is applied to the non-influencing fasteners usingan X-pattern sequence to minimize the application of any stresses.

Thermal sensors may be used to measure the thermal interface gapachieved by method 700. If the desired thermal interface gap is notachieved, then the unit may be simply reworked by removing the heatsink/load arm assembly from the load frame/spring assembly, and cleaningthe thermally conductive material from the semiconductor chip, andreturning to step 740.

As noted above, in order to provide greater performance, it willeventually be necessary to increase processor chip powers beyond thepoint where forced air-cooling is feasible as a solution. To meet thisincreased cooling demand, a liquid-based cooling system is providedherein, with a liquid-cooled cold plate physically coupled to eachprimary heat generating component to be cooled. FIG. 8 is a depiction ofthe electronics drawer component layout of FIG. 2, shown with such acooling system.

More particularly, FIG. 8 depicts one embodiment of an electronicsdrawer 813 component layout wherein one or more air moving devices 811provide forced air flow 815 to cool multiple components 812 withinelectronics drawer 813. Cool air is taken in through a front 831 andexhausted out a back 833 of the drawer. The multiple components to becooled include multiple processor modules to which liquid-cooled coldplates 820 (of a liquid-based cooling system) are coupled, as well asmultiple arrays of memory modules 830 (e.g., dual in-line memory modules(DIMMs)) and multiple rows of memory support modules 832 (e.g., DIMMcontrol modules) to which air-cooled heat sinks are coupled. In theembodiment illustrated, memory modules 830 and the memory supportmodules 832 are partially arrayed near front 831 of electronics drawer813, and partially arrayed near back 833 of electronics drawer 813.Also, in the embodiment of FIG. 8, memory modules 830 and the memorysupport modules 832 are cooled by air flow 815 across the electronicsdrawer.

The illustrated liquid-based cooling system further includes multiplecoolant-carrying tubes connected to and in fluid communication withliquid-cooled cold plates 820. The coolant-carrying tubes comprise setsof coolant-carrying tubes, with each set including (for example) acoolant supply tube 840, a bridge tube 841 and a coolant return tube842. In this example, each set of tubes provides liquid coolant to aseries-connected pair of cold plates 820 (coupled to a pair of processormodules). Coolant flows into a first cold plate of each pair via thecoolant supply tube 840 and from the first cold plate to a second coldplate of the pair via bridge tube or line 841, which may or may not bethermally conductive. From the second cold plate of the pair, coolant isreturned through the respective coolant return tube 842.

FIG. 9 depicts in greater detail an alternate electronics drawer layoutcomprising eight processor modules, each having a respectiveliquid-cooled cold plate of a liquid-based cooling system coupledthereto. The liquid-based cooling system is shown to further includeassociated coolant-carrying tubes for facilitating passage of liquidcoolant through the liquid-cooled cold plates and a header subassemblyto facilitate distribution of liquid coolant to and return of liquidcoolant from the liquid-cooled cold plates. By way of specific example,the liquid coolant passing through the liquid-based cooling subsystem ischilled water.

As noted, various liquid coolants significantly outperform air in thetask of removing heat from heat generating electronic components of anelectronics system, and thereby more effectively maintain the componentsat a desireable temperature for enhanced reliability and peakperformance. As liquid-based cooling systems are designed and deployed,it is advantageous to architect systems which maximize reliability andminimize the potential for leaks while meeting all other mechanical,electrical and chemical requirements of a given electronics systemimplementation. These more robust cooling systems have unique problemsin their assembly and implementation. For example, one assembly solutionis to utilize multiple fittings within the electronics system, and useflexible plastic or rubber tubing to connect headers, cold plates, pumpsand other components. However, such a solution may not meet a givencustomer's specifications and need for reliability.

Thus, presented herein is a robust and reliable liquid-based coolingsystem specially preconfigured and prefabricated as a monolithicstructure for positioning within a particular electronics drawer.

FIG. 9 depicts is an isometric view of one embodiment of an electronicsdrawer and monolithic cooling system, in accordance with an aspect ofthe present invention. The depicted planar server assembly includes amulti-layer printed circuit board to which memory DIMM sockets andvarious electronic components to be cooled are attached both physicallyand electrically. In the cooling system depicted, a supply header isprovided to distribute liquid coolant from a single inlet to multipleparallel coolant flow paths and a return header collects exhaustedcoolant from the multiple parallel coolant flow paths into a singleoutlet. Each parallel coolant flow path includes one or more cold platesin series flow arrangement to cool one or more electronic components towhich the cold plates are mechanically and thermally coupled. The numberof parallel paths and the number of series-connected liquid-cooled coldplates depends, for example on the desired device temperature, availablecoolant temperature and coolant flow rate, and the total heat load beingdissipated from each electronic component.

More particularly, FIG. 9 depicts a partially assembled electronicssystem 913 and an assembled liquid-based cooling system 915 coupled toprimary heat generating components (e.g., including processor dies) tobe cooled. In this embodiment, the electronics system is configured for(or as) an electronics drawer of an electronics rack, and includes, byway of example, a support substrate or planar 905, a plurality of memorymodule sockets 910 (with the memory modules (e.g., dual in-line memorymodules) not shown), multiple rows of memory support modules 932 (eachhaving coupled thereto an air-cooled heat sink 934), and multipleprocessor modules (not shown) disposed below the liquid-cooled coldplates 920 of the liquid-based cooling system 915.

In addition to liquid-cooled cold plates 920, liquid-based coolingsystem 915 includes multiple coolant-carrying tubes, including coolantsupply tubes 940 and coolant return tubes 942 in fluid communicationwith respective liquid-cooled cold plates 920. The coolant-carryingtubes 940, 942 are also connected to a header (or manifold) subassembly950 which facilitates distribution of liquid coolant to the coolantsupply tubes and return of liquid coolant from the coolant return tubes942. In this embodiment, the air-cooled heat sinks 934 coupled to memorysupport modules 932 closer to front 931 of electronics drawer 913 areshorter in height than the air-cooled heat sinks 934′ coupled to memorysupport modules 932 near back 933 of electronics drawer 913. This sizedifference is to accommodate the coolant-carrying tubes 940, 942 since,in this embodiment, the header subassembly 950 is at the front 931 ofthe electronics drawer and the multiple liquid-cooled cold plates 920are in the middle of the drawer.

Referring more particularly to FIGS. 9 & 10A, liquid-based coolingsystem 915 comprises a preconfigured monolithic structure which includesmultiple (pre-assembled) liquid-cooled cold plates 920 configured anddisposed in spaced relation to engage respective heat generatingelectronic components. Each liquid-cooled cold plate 920 includes, inthis embodiment, a liquid coolant inlet 1002 (see FIG. 10A) and a liquidcoolant outlet 1004, as well as an attachment subassembly 1020 (i.e., acold plate/load arm assembly). In a similar manner to the heat sinkattachment approach of FIGS. 3-7, each attachment subassembly 1020 isemployed to couple its respective liquid-cooled cold plate 920 to theassociated electronic component to form the cold plate and electroniccomponent assemblies depicted in FIG. 9. Alignment openings (i.e.,thru-holes) 1010 are provided on the sides of the cold plate to receivealignment pins 332 (FIG. 3) or positioning dowels 1120 (FIG. 11) duringthe assembly process, as described further in the above-incorporatedpatent application entitled “Method of Assembling a Cooling System for aMulti-Component Electronics System”. Additionally, connectors (or guidepins) 1022 are included within attachment subassembly 1020 whichfacilitate use of the attachment assembly, as explained below withreference to FIGS. 11 & 12. Note that load arms 1024 of connectorassembly 1020 are also shown in FIG. 10A.

As shown in FIGS. 9 & 10B, header subassembly 950 includes two liquidmanifolds, i.e., a coolant supply header 952 and a coolant return header954, which in one embodiment, are coupled together via supportingbrackets 1030. In the monolithic cooling structure of FIG. 9, thecoolant supply header 952 is metallurgically bonded in fluidcommunication to each coolant supply tube 940, while the coolant returnheader 954 is metallurgically bonded in fluid communication to eachcoolant return tube 952. A single coolant inlet 951 and a single coolantoutlet 953 extend from the header subassembly for coupling to theelectronics rack's coolant supply and return manifolds (not shown).

FIGS. 9 & 10C depict one embodiment of the preconfigured,coolant-carrying tubes. In addition to coolant supply tubes 940 andcoolant return tubes 942, bridge tubes or lines 941 are provided forcoupling, for example, a liquid coolant outlet of one liquid-cooled coldplate to the liquid coolant inlet of another liquid-cooled cold plate toconnect in series fluid flow the cold plates, with the pair of coldplates receiving and returning liquid coolant via a respective set ofcoolant supply and return tubes. In one embodiment, the coolant supplytubes 940, bridge tubes 941 and coolant return tubes 942 are eachpreconfigured, semi-rigid tubes formed of a thermally conductivematerial, such as copper or aluminum, and the tubes are respectivelybrazed, soldered or welded in a fluid-tight manner to the headersubassembly and/or the liquid-cooled cold plates. The tubes arepreconfigured for a particular electronics system to facilitateinstallation of the monolithic structure in engaging relation with theelectronics system.

To summarize, a cooling system such as disclosed in connection withFIGS. 9-10C advantageously comprises a monolithic structurepreconfigured for actively cooling multiple heat generating electroniccomponents of an electronics system. The monolithic structure includesmultiple liquid-cooled cold plates disposed in spaced relation, witheach liquid-cooled cold plate of the multiple liquid-cooled cold platesbeing configured and positioned to couple to a respective heatgenerating electronic component of the multiple heat generatingelectronic components to be cooled. A plurality of coolant-carryingtubes are metallurgically bonded in fluid communication with multiplecold plates and with a liquid-coolant header subassembly. Theliquid-coolant header subassembly includes a coolant supply headermetallurgically bonded in fluid communication with the multiple coolantsupply tubes and a coolant return header metallurgically bonded in fluidcommunication with multiple coolant return tubes. When in use, themultiple liquid-cooled cold plates are coupled to respective heatgenerating electronic components and liquid coolant is distributedthrough the header subassembly and coolant-carrying tubes to the coldplates for removal of heat generated by the electronic components.

Advantageously, the configuration depicted routes coolant in such amanner as to provide multiple parallel paths through multipleseries-connected liquid-cooled cold plates. This configurationfacilitates maintaining a desired drawer level pressure drop and adesired electronic component level temperature rise. The monolithicstructure is mounted to, for example, the planar circuit board orstiffener via brackets mounted to the header subassembly and a coldplate to electronic component attachment subassembly (see FIGS. 11 & 12)similar to the mounting mechanism depicted and described in detail abovein connection with FIGS. 3-7. The cooling system embodiment depicted isdesigned for direct attachment of the liquid-cooled cold plates to theelectronics component to be cooled, which may include one or more baredies, thereby eliminating the traditional lid and second thermalinterface material.

FIGS. 11 & 12 depict one embodiment of a liquid-cooled cold platedirectly attached to an electronic component comprising multiple baredies residing on a common carrier. As best shown in FIG. 12, the coldplate includes a cold plate base 1200, an active heat transfer region orstructure 1220 and a cold plate lid 1210 having, for example, a coolantinlet 1002 and coolant outlet 1004. The heat generating electroniccomponent 1230 includes, in this example, a carrier 1236 supporting twoprimary heat generating dies 1232 and two secondary heat generating dies1234, each of which is assumed to be a bare die. Additionally, dies 1232are assumed to generate greater heat than dies 1234. In the illustratedembodiment, the active heat transfer structure 1220 of the cold plate isconfigured to reside only over the primary heat generating dies 1232 formore active cooling of the dies compared with dies 1234.

Electronic component 1230 is disposed within a central opening in aloading frame 1100. When in use, loading frame 1100 is affixed to theelectronic system's printed circuit board or planar, and sets theposition for the loading and cooling hardware. Carrier 1236 ofelectronic component 1230 is assumed to be mechanically and electricallycoupled to the printed circuit board as well. A thermal interfacematerial, such as a thermally conductive gel, is disposed between thebare die back sides and the cold plate's contacting surface, whichcontacts the bare dies. Again, the active heat transfer structure 1220of the cold plate is aligned (in this example) only over the highpowered bare dies 1232 (e.g., processor dies). This embodiment seeks tocool the higher power chips preferentially in order to maintain adesired junction temperature in all of the devices being cooled.

The attachment subassembly again includes a pair of load springs 1110connected to load frame 1100. Load frame 1100 is preferably made of analloy material chosen for its low creep properties, such as Zamak 8,while load springs 1110 are preferably made of an alloy material chosenfor its high tensile strength properties, such as a high strength musicwire. Although two load springs 1110 are shown in FIGS. 11 & 12, thoseskilled in the art will appreciate that the present invention may bepracticed with any number of load springs 1110. Load frame 1100 is againmounted to the printed circuit board via fasteners, such as the screwsdescribed above in connection with the embodiment of FIG. 5. Positioningdowels 1120 on either side of the frame engage respective thru-holes1301 (FIG. 13) on either side of the cold plate base 1200. One or morenon-influencing fasteners 1130 are used to secure the cold plate/loadarm assembly to the load frame assembly. By way of example, fournon-influencing fasteners 1130 are mounted on load frame 1100. Thenon-influencing fasteners 1130, which in one embodiment are threadedinto respective bosses of load frame 1100, lock the cold plate inposition without influencing the position of the cold plate in a mannersimilar to that described above in connection with FIGS. 3-6A.

The attachment subassembly again includes load arms 1024 hingedlyconnected via pins 1225 to a U-channel load bracket 1020, which hasopenings to accommodate load transfer block fasteners 1022. Fasteners1022 are threaded at their distal ends to engage respective threadedopenings 1226 in an upper surface of the cold plate base. Load transferblock fasteners 1022 further function as load bracket retaining dowelsin this embodiment. A load transfer block 1221 is disposed below theload bracket 1020 and a load actuation screw 1105 applies compressiveforce to load transfer block 1221, which in turn applies a compressiveload to the cold plate, and hence to the back side of the bare die ofthe electronic component to ensure a desired thermal interface materialthickness, and thus a favorable thermal interface resistance between thebare dies and the contacting surface of the cold plate. As is known, thethermal resistance of the thermal interface material is inverselyproportional to the material's thickness. Advantageously, the cold platebase and load transfer block are configured to distribute loadingpressure across the raised, planar upper surface of the cold plate base.

FIGS. 13 & 13A depict one detailed embodiment of cold plate base 1200.As shown, base 1200 is again configured with active heat transferstructure 1220 extending only over a portion thereof. Within the activeheat transfer structure 1220, multiple parallel channels 1300 aredisposed for passing liquid coolant therethrough. Dowel receivingthru-holes 1301 are provided on either side of the active heat transferstructure for engaging positioning dowels 1120 (FIG. 12). Further,threaded openings 1226 are provided in the upper surface of the coldplate base 1200 and are located to receive respective load transferblock fasteners 1022 (FIG. 12), as described above. A brazing pocket1310 is also shown in FIG. 13A for facilitating brazing of cold platelid 1210 (FIG. 12) to cold plate base 1200. Base cutout areas 1320,which are provided for mass reduction, result in the raised, planarupper surface configuration (when the cold plate lid is attached)illustrated in FIG. 13.

FIGS. 14 & 14A depict an alternate embodiment of a cold plate toelectronic component attachment subassembly. In FIGS. 14 & 14A a singlecold plate, electronic component, and attachment assembly is shown. Thestructure includes a cold plate 1410 mounted to an electronic component1420 via an attachment subassembly 1430, which includes two loading arms1432 hingedly mounted to a base 1434 coupled to the substrate (notshown). Laminated spring plates 1436 and an actuation screw 1438 areemployed to adjust pressure against cold plate 1410, and hence couplingpressure between cold plate 1410 and electronic component 1420, which isassumed to plug into a corresponding socket in a preconfiguredsupporting substrate or motherboard. The electronic component 1420 isshown to be a lidded electronic module and may comprise any electroniccomponent to be cooled, including one or more bare die. Features ofattachment subassembly 1430 depicted in FIGS. 14 & 14A may be betterunderstood with reference to the above-incorporated patent applicationentitled “Heat Sink Apparatus for Applying a Specified Compressive Forceto an Integrated Circuit Device”, U.S. Ser. No. 11/460,334, filed Oct.10, 2006.

Briefly described, actuation of the attachment subassembly is providedby the fixed travel of the actuation screw 1438 through, for example,two laminated spring plates 1436. Spring plates 1436 reside atop aU-channel structure which physically contacts and is attached to theupper portion of the cold plate. The upper portion of the cold plate isshown to have inlet 1412 and outlet 1414, and is assumed to be brazed toa lower portion of the cold plate. The lower portion of the cold platehas the necessary fin structures and coolant reservoir (e.g., see FIGS.13 & 13A).

Once an actuation load is applied, load arms 1432 on the sides of theattachment subassembly are locked down and the load is maintained. Asone example, the load required to actuate one embodiment of a hybrid LGA(land grid array) is 200 pounds. This actuation load provides multiplefunctions, including: maintaining thermal interface material gapthickness (between the cold plate and electronic component) (see layer1945 in FIG. 19) to approximately 30-50 microns; providing actuation ofindividual electrical contacts on the hybrid LGA socket; providing forelectronic component tolerances; and providing enough load on theactuation hardware with the needed compliance of the cooling systemassembly (e.g., the monolithic cooling system assembly depicted in FIG.9 above).

Each electronic component is allowed to vary in height by a giventolerance, and a certain amount of height difference is allowed foractuation of the fastening hardware. Compliance in the cooling system isprovided by specifically designing the tube interconnections betweenseries-connected cold plates to allow for the necessary actuationpressure to be applied to the individual electrical contacts. It isassumed herein that the coolant-carrying tube layout for interconnectingat least two liquid-cold cold plates in series-fluid communicationemploys one or more rigid tubes. These one or more rigid tubes arelargely non-compliant and create a robust structure that can be brazedtogether to mitigate the possibility of leaks, as noted above. Thecoolant-carrying tube layout is configured herein to provide sufficientcompliance to accommodate, for example, misalignment of electroniccomponents, as well as tolerance differences. Actuation hardware isprovided (e.g., FIGS. 14 & 14A) to mate the electronic component to thecorresponding (for example, hybrid LGA) socket on the preconfiguredmotherboard as it simultaneously squeezes thermal grease between thecold plate and lid of the electronic component to generate the thermalinterface (i.e., TIM 2) therebetween. This dual function of theattachment subassembly is described in the above-incorporated co-pendingapplication entitled “Heat Sink Apparatus for Applying a SpecifiedCompressive Force to an Integrated Circuit Device”.

FIG. 15 depicts in detail an alternate electronics drawer layoutcomprising sixteen processor modules, each having a respectiveliquid-cooled cold plate of a liquid-based cooling system coupledthereto. The liquid-based cooling system is shown to include associatedcoolant-carrying tubes for facilitating passage of liquid coolantthrough the liquid-cooled cold plates and a header subassembly tofacilitate distribution of liquid coolant to and return of liquidcoolant from the liquid-cooled cold plates. By way of example, theliquid-cooled cold plates are configured in multiple coolingsubassemblies, each cooling subassembly comprising four liquid-cooledcold plates connected in series-fluid communication between a coolantsupply tube 1540 and a coolant return tube 1542 as shown in FIG. 16.

Presented herein is a robust and reliable liquid-based cooling systemand method of configuring a monolithic structure for positioning withina particular electronics drawer configuration. FIG. 15 depicts anisometric view of one embodiment of an assembled electronics system 1513and an assembled liquid-based cooling system 1515 coupled to primaryheat-generating components (e.g., including processor dies) to becooled. In this embodiment, the electronics system is configured for (oras) an electronics drawer of an electronics rack, and includes, by wayof example, a support substrate or motherboard 1505, a plurality ofmemory modules 1510, and multiple processor modules 1610 (see FIG. 16A)disposed below the liquid-cooled cold plates 1520 of the liquid-basedcooling system 1515.

In addition to liquid-cooled cold plates 1520, liquid-based coolingsystem 1515 includes multiple coolant-carrying tubes, including coolantsupply tubes 1540 and coolant return tubes 1542 in fluid communicationwith respective cooling subassemblies. The coolant-carrying tubes 1540,1542 are also connected to a header (or manifold) subassembly 1550 whichfacilitates distribution of liquid coolant to the coolant supply tubesand return of liquid coolant from the coolant return tubes. In thisembodiment, there are four cooling subassemblies, each comprising fourliquid-cooled cold plates 1520 coupled in series-fluid communicationemploying multiple rigid tubes 1600 as shown in FIG. 16. The multiplerigid tubes 1600 couple, for example, the coolant outlet of oneliquid-cooled cold plate to the coolant inlet of another liquid-cooledcold plate of the four series-connected liquid-cooled cold plates.

As noted, disclosed herein (in one aspect) are methods for configuringcoolant-carrying tube layouts for interconnecting in-series multipleliquid-cooled cold plates using substantially rigid tubes, while stillproviding sufficient compliance to avoid an unacceptable reduction inactuation load resulting from the use of the rigid tubes in thecoolant-carrying tube layout. The methods disclosed herein employsubstantially rigid metallic tubing and provide sufficient compliance tomeet displacement requirements for actuation loading, as well asrelative positional tolerances for a given preconfigured cooling system.Tubing layout is designed to satisfy the desired fluidic arrangementamong the liquid-cooled cold plates, then (in one embodiment) finiteelement analysis is done to determine stresses on the tubing, and lossof actuation load resulting therefrom.

As a specific example, for an LGA electrical socket, 30 grams perconnection is required for an acceptable and reliable electricalconnection. The actuation hardware is designed to provide this level ofactuation, plus a small margin in the absence of the coolant-carryingtubing. Three-dimensional numerical analysis of stress and strain on acreated coolant-carrying tube layout is employed to determine (forexample) relative planar tilt on an associated liquid-cooled cold plateresulting from an applied actuation load, as well as the reduction inavailable load to the LGA connections. The resulting information is thenused to determine whether lengthened tube portions are needed, orwhether to add one or more horizontal bends to interconnecting tubing,with the goal of retaining co-planarity of the electronic component andLGA socket under load, and to reduce the loss of actuation loadresulting from the coolant-carrying tube layout. If necessary, thecoolant-carrying tube layout is reconfigured, and the reconfigured tubelayout is fed back for stress analysis until an acceptable level ofco-planarity under load is achieved, and there is a low loss ofactuation load (e.g., less than 3% of available actuation load).

FIGS. 17A-17C depict one detailed embodiment of processing for creatingand verifying a coolant-carrying tube layout for a cooling subassemblysuch as depicted in FIGS. 15-16A.

The process begins with ascertaining a preconfigured electronics systemlayout, e.g., a fixed motherboard layout, including DIMM positions,electronic component positions, etc. 1700. Hydraulic calculations areemployed to determine a particular parallel-series arrangement for thecooling system connections. It is assumed herein that at least two coldplates are connected in-series to form a cooling subassembly, and that(in one embodiment) coolant is fed in parallel to multiple coolingsubassemblies of the cooling system. An example of a preconfiguredelectronics system layout is depicted in FIG. 15. Next, a basiccoolant-carrying tube layout is created, taking into account designrules of tube outside diameter, minimum bend radius, and any locationsthat must be accessible on the electronics system or attachmentsubassembly when the cooling system is coupled thereto 1702. By way ofexample, interconnect tubing might include one or more horizontal bendsand comprise copper tubing with one-quarter inch OD and 0.030 inch wallthickness, annealed temper and a design limit of 10,000 psi stress.Alternatively, the basic tube layout may be a simplest layout“connecting the dots”; that is, a simplest layout connecting the coldplate outlets and inlets of the cold plates in series-fluidcommunication within the cooling subassembly (with or without ahorizontal bend).

Knowing the available actuation load and required final load on eachelectrical connection, a simplified structural analysis or simulation iscreated to analyze the rigid body reaction force applied by thecoolant-carrying tube layout of the cooling subassembly 1704. By way ofspecific example, the available actuation load might be 200 pounds, andthe required final load on each electrical connection 30 grams. Anexample of a simplified analysis is a simplified three-dimensionalnumerical analysis of stress and strain employing, for example, a finiteelement analysis simulation product. One example of a finite elementanalysis simulation product which could be employed in connection withthe processing described herein is a structure simulation product, suchas those offered by ANSYS, Inc. of Cannonsburg, Pa., USA.

The simplified analysis is performed on the cooling subassembly todetermine stress in the rigid tubing serially interconnecting theliquid-cooled cold plates, as well as loss of actuation load resultingfrom the coolant-carrying tube layout created 1706. The simplifiedstructural analysis includes the liquid-cooled cold plates and theinterconnecting tubing, as illustrated in FIG. 16. By way of example,only the subset of liquid-cooled cold plates within a particular coolingsubassembly is included in the analysis. The purpose of this calculationis to determine the required force applied to a cold plate that wouldresult in an amount of deflection commensurate with the anticipatedlocational tolerance among the electronic component surfaces to becontacted by the cold plates. To facilitate this analysis, all but oneof the cold plates (e.g., cold plate 1820′ of FIG. 18) are constrainedagainst any motion, while the force is applied in a downward directionto the remaining cold plate 1820, as illustrated in FIG. 18. Theresulting deformation is studied, and the magnitude of the forceadjusted, until the cold plate is displaced a distance equal to thetotal tolerance among the electronic component heights within theelectronics system. This analysis may be done on the “worst case” coldplate, that is, the cold plate chosen to apply the force and studydisplacement should be the one that requires the largest amount of forceto get the desired displacement. The significance of this required forceis that it is the sum total of the force in the Z-direction required toovercome the reactionary forces applied by the rigid tubinginterconnecting the liquid-cooled cold plates in series-fluidcommunication. This reactionary force reduces the amount of forceavailable to make electrical connections between the electroniccomponents to be cooled and the preconfigured motherboard.

As shown in FIG. 18, the simplified analysis applies a load 1810 to one1820 of the series-connected cold plates 1820, 1820′, withoutconsidering the electrical connections present. The load is applieduntil the total misalignment due to tolerance in actuation is achievedby varying the load. For example, it may require 4.5 pounds to deflectliquid-cooled cold plate 1820 by 0.9 mm in the direction normal to thethermal contact surface (i.e., the Z-direction in this example).

Processing then determines if the stresses on the rigid tubing and theloss of actuation load are acceptable 1708. If stress (in this example)exceeds 10,000 psi in the rigid tube, or if the loss of actuation loadexceeds (for example) 5 pounds, then the actuation load is unacceptable,and processing determines whether any distance between a cold plateoutlet and a cold plate inlet in the cooling subassembly can beincreased; that is, whether the outlet or inlet on a particular coldplate can be moved 1710. This inquiry takes into account anypredetermined design rules regarding accessibility of one or morelocations of the assembly once the cooling system is coupled to theelectronics system. If “yes”, then the outlet or inlet of one or moreliquid-cooled cold plates of the cooling subassembly is moved to, forexample, maximize a tube path length of an outlet to inlet tube 1712interconnecting in-series the two cold plates. This results in areconfigured coolant-carrying tube layout, which then undergoessimplified stress analysis as described above in connection with step1706.

If the distance between cold plate connections cannot be increased, thenprocessing examines the stress in the interconnect tubing looking for atleast one tube with high stress 1714. For example, stresses in the bendsor elbows should not exceed 10,000 psi. The bends or elbows are theregions where stress concentration is typically highest. Determinationis then made whether stress in any tubing is higher than the yieldstrength of the tubing material 1716. As one example, the yield strengthof a fully annealed temper copper ACR (i.e., air conditioning andrefrigeration) tubing is about 10,000 psi. If tube stress in at leastone tube is greater than the yield strength, then the at least one tubeis lengthened both before and after the bend having the high stress todecrease the reactionary force and the total angular displacement 1718.It is noted that the stress will tend to be highest at the inside of thebend. A longer tube in the section of the bend closer to the displacedliquid-cooled cold plate has the effect of lengthening the lever arm(i.e., the tube portion in bending 1823 (FIG. 18)) and thus reducing theangular displacement on the high stress area, resulting in a lowerstrain, and thus lower stress. On the other hand, lengthening of thetube section on the other side of the bend from the displaced cold platehas the effect of reducing reactionary force on the bend supplied bywhat is in effect the tube portion in torsion (i.e., that tube portion1825 (FIG. 18) acting as a torsional spring).

Assuming that the stresses in the interconnecting tubes are lower thanthe yield strength for the tubing material, then processing examines thetilt of one or more cold plates of the cooling subassembly employing thesimplified analysis or simulation 1720. If the reactionary force on thetubes (in opposite corners in the embodiment of FIG. 18) are unequal,then the resultant force acting on the cold plate will not be centeredon the centroid of the cold plate, thus, any dowels positioning the coldplate relative to the electronics system may bind, or loading on theelectrical connections coupling the associated electronic component tothe sockets of the motherboard may not be uniform. If one tube suppliesmore reactionary force than another tube, then the tube supplying thehigher reactionary force should be modified. That is, the tubeassociated with the higher corner of a tilted cold plate is selected forreconfiguration to achieve greater compliance.

Processing determines whether a lengthened tube portion in torsionresults in a substantially lower reactionary force on the cold plate1722. This is based on the simplified analysis and the judgment of thesystem designer, given the cross-section and strength of tube materialemployed. If the tube is flexible in torsion, then it would bebeneficial to load as much of the tubing as possible in torsion toreduce reactionary loads on the liquid-cooled cold plates. If alengthened tube portion in torsion does reduce the reactionary load,then the coolant-carrying tube layout is reconfigured to allow for thelengthened tube portion in torsion 1724 and the reconfigured tube layoutundergoes the simplified analysis. (If necessary, “service loops” areadded to increase the tube length by inserting one or more horizontalbends into the tube length (e.g., one or more 90° horizontal bends inthe tube)). If a lengthened tube portion in torsion does not reduce thereactionary load, then processing determines whether a lengthened tubesection in bending results in a substantially lower reactionary load onthe cold plate 1726. This again can be based on the system designer'sjudgment as to the strength of the tube in bending using the knowncross-sectional shape and area moment of inertia, as well as the lengthof the tube.

It may be beneficial to lengthen tubes that are experiencing a bendingcondition. If “yes”, then the tube layout is reconfigured to allow forthe longer tube portion in bending 1728 and the reconfigured tube layoutis returned to undergo the simplified stress analysis. If the lengthenedtube portion in bending does not result in a substantially lowerreactionary force on the cold plate, then processing determines whetherone or more horizontal bends can be added to the identifiedcoolant-carrying tube having high stress 1727, and if “yes”, one or morehorizontal bends are added 1729 and the reconfigured tube layoutundergoes the simplified analysis. Otherwise, processing detects anerror 1730 and one or more design rules or original assumptions aremodified. For example, the tubing outside diameter could be reduced,followed by restart of the design process, or the assumed configurationof the electronics system (e.g., electronics drawer layout) could bemodified and the design process restarted, or the tube material employedcould be changed, and the design process restarted. This is necessarywhen all possibilities for improving the tube layout within the givenconstraints have been exhausted. In this case, one or more designassumptions need to change so that a compromise can be reached.

Returning to inquiry 1708, once stress and loss of actuation load areacceptable, then a more detailed three-dimensional numerical analysis ofstress and strain is performed, including the electrical connectionloads between the associated electronic components and the motherboardsockets receiving the electronic components 1732. As one example, thisdetailed analysis includes the cold plate, as well as the hardwareillustrated in FIG. 19. This hardware includes, in the illustratedembodiment, a stiffener 1905, an insulator 1910, a supportingmotherboard 1915, a perimeter support/seal 1920, an LGA interposer 1925,a bare integrated circuit die and supporting substrate 1930, a firstthermal interface material 1935, an electronic component lid 1940 and asecond thermal interface material 1945. The purpose of this moredetailed simulation is to study the deformation of the cold plate andelectronic component, and to determine the force making each of thehundreds (or thousands) of electrical connections between the electroniccomponent and the motherboard.

More particularly, the structure of the electronic component includes alarge number (possibly thousands) of metal pads on the bottom surface ofthe component designed to make electrical connection to similar pads onthe motherboard. The LGA interposer includes a structure holdingelectrically conductive springs, one for each set of metal pads, with acontacting surface on either side. There is a minimum amount of forcerequired to achieve a reliable and secure electrical connection betweeneach component pad, interposer and board pad. To determine the effect ofthe tubes on the electrical connections, the simplified model isanalyzed at the tube to cold plate connection to see what the resultantforce of the tube on the cold plate is under the conditions listed abovefor the simplified simulation. These forces are then applied to thecorresponding locations on the cold plate in the detailed simulation.The detailed simulation is then analyzed to determine whether thereexists an acceptable load on each of the electrical connections despitethe influence of the interconnecting tubes.

If the electrical connection loads are unacceptable 1734, then theacceptable limits for the simplified analysis are redefined 1736. If thesimplified analysis indicates acceptable reactionary forces and thedetailed analysis indicates electrical connections below the specifiedacceptable force level, then the success criteria for the simplifiedanalysis must be made more stringent. After redefining the simplifiedsimulation, processing returns to inquiry 1710.

Assuming that the electrical connection loading is acceptable,processing performs computational fluid dynamics analysis on theproposed final design to ensure that the pressure drop and flowcharacteristics have not changed beyond acceptable limits, as definedinitially in step 1700. Examples of computational fluid dynamicsproducts which could be employed to facilitate the analysis are Fluentand/or Ansys CFX, both offered by ANSYS, Inc. of Cannonsburg, Pa., USA.Inquiry is made whether the pressure drop and flow characteristics meetthe initial criteria 1740. If “no”, then the tube layout is reconfiguredto reduce the number of bends, or to shorten one or more path lengths,possibly invoking changing placement of a fluid outlet or inlet of therespective cold plates 1742. The reconfigured tube layout is then fedback to the simplified analysis for repeating of the processingdescribed above. Assuming that the pressure drop and flowcharacteristics are acceptable, then the final design is achieved andsaved 1744.

To summarize, those skilled in the art will note from the abovedescription that provided herein are various methods for configuring acooling subassembly for an electronics system. By way of example, onemethod is presented which includes: obtaining a coolant-carrying tubelayout for interconnecting multiple liquid-cooled cold plates inseries-fluid communication for cooling multiple heat-generatingelectronic components of an electronics system, the interconnectedmultiple liquid-cooled cold plates being a cooling subassembly, themultiple heat-generating electronic components to be plugged in fixedspaced relation into corresponding sockets on a supporting motherboard,and the coolant-carrying tube layout comprising at least one rigidcoolant-carrying tube; determining for the coolant-carrying tube layoutif stress in one or more rigid coolant-carrying tubes thereof exceeds apredetermined acceptable level, and determining if loss of actuationload on a heat-generating electronic component being cooled by thecooling subassembly exceeds an acceptable loss threshold, and if eitheris true, identifying at least one tube of the coolant-carrying tubelayout having high stress, the at least one tube comprising at least oneof a tube portion in torsion and a tube portion in bending when a forceis applied to at least one liquid-cooled cold plate of the coolingsubassembly, and reconfiguring the at least one tube having high stressto produce a reconfigured coolant-carrying tube layout and repeating thedetermining for the reconfigured coolant-carrying tube layout;performing analysis on the cooling subassembly and the multipleheat-generating electronic components of the electronics system once thecoolant-carrying tube layout experiences tube stress below the definedacceptable level and a loss of actuation load on the multipleheat-generating electronic components below the acceptable lossthreshold, the analysis including employing the available actuation loadon at least part of the cooling subassembly and evaluating electricalconnection loading between at least one heat-generating electroniccomponent being cooled thereby and the supporting motherboard into whichthe at least one heat-generating electronic component electricallyconnects in fixed spaced relation; and saving or otherwise employing thecoolant-carrying tube layout as a final design for interconnecting inseries-fluid communication the multiple liquid-cooled cold plates of thecooling subassembly if the electrical connection loading is above anacceptable minimum level.

As an enhancement, the determining includes performing simplifiedanalysis (e.g., a simplified simulation) on the cooling subassembly todetermine for the coolant-carrying tube layout stress on the at leastone coolant-carrying tube interconnecting the multiple liquid-cooledcold plates and loss of actuation load on at least one heat-generatingelectronic component of the multiple heat-generating electroniccomponents resulting from the coolant-carrying tube layout. Thesimplified analysis is performed without evaluating electricalconnection loading for the multiple heat-generating electroniccomponents into the supporting motherboard. More particularly, thesimplified analysis includes performing a simplified three-dimensionalnumerical analysis of stress and strain on the cooling subassembly,which in one example, is a finite element analysis simulation of thecooling subassembly to evaluate stress on the at least onecoolant-carrying tube interconnecting the multiple liquid-cooled coldplates and loss of actuation load on the at least one heat-generatingelectronic component of the multiple heat-generating electroniccomponents resulting from the coolant-carrying tube layout. The moredetailed analysis on the cooling subassembly and multipleheat-generating electronic components includes performing athree-dimensional numerical analysis of stress and strain on the coolingsubassembly and the multiple heat-generating electronic components,including the available actuation load on the cooling subassembly toevaluate electrical connection loading on the multiple heat-generatingelectronic components and the supporting motherboard into which themultiple heat-generating electronic components electrically connect. Inone embodiment, a finite element analysis simulation of the coolingsubassembly and multiple heat-generating electronic components isperformed to evaluate electrical connection loading on the multipleheat-generating electronic components and the supporting motherboardinto which the multiple heat-generating electronic componentselectrically connect.

The reconfiguring can include, for example, reconfiguring one or moretubes having high stress by introducing one or more bends into thetube(s), lengthening a tube portion in torsion thereof, or lengthening atube portion in bending thereof. As a more specific example, thereconfiguring can include determining whether a lengthened tube portionin torsion for the at least one tube having high stress produces atleast one of a lower stress on the at least one tube or a lowerreactionary force on at least one heat-generating electronic componenthaving loss of actuation load beyond the acceptable loss threshold, andif so, reconfiguring the coolant-carrying tube layout to include thelengthened tube portion in torsion and repeating the determining for thereconfigured coolant-carrying tube layout. Otherwise, the reconfiguringcan include determining whether a lengthened tube portion in bending forthe at least one tube having high stress results in at least one of alower stress on the at least one tube or a lower reactionary force onthe at least one heat-generating electronic component having loss ofactuation load beyond the acceptable loss threshold, and if so,reconfiguring the coolant-carrying tube layout to include the lengthenedtube portion in bending, and repeating the determining for thereconfigured coolant-carrying tube layout.

Additionally, or alternatively, the reconfiguring can includedetermining whether a distance between an outlet and an inlet for atleast two series-connected liquid-cooled cold plates of the multipleliquid-cooled cold plates can be increased, and if so, redefining alocation of at least one of the outlet and the inlet for the at leasttwo series-connected liquid-cooled cold plates to maximize a path lengthfor a coolant-carrying tube interconnecting the outlet and the inlet ofthe at least two series-connected liquid-cooled cold plates, whilemaintaining any predefined constraints on the location of the inlet andthe outlet for the at least two series-connected liquid-cooled coldplates. Identifying one or more tubes having high stress can includeexamining tilt of the cold plates when a force is applied to one or moreof the cold plates. For at least one liquid-cooled cold plate havingtilt, a tube associated with a highest corner thereof is identified as atube having high stress.

As a further enhancement, the method can include performingcomputational fluid dynamics analysis on the cooling subassembly todetermine if pressure drop and flow distribution are acceptable throughthe cooling subassembly prior to selecting the coolant-carrying tubelayout as the final design. The method includes reconfiguring thecoolant-carrying tube layout if pressure drop or flow distributionthrough the cooling subassembly is unacceptable. This reconfiguringincludes at least one of reducing a number of bends in thecoolant-carrying tube layout or reducing a length of at least one tubepath length in the coolant-carrying tube layout to create a newcoolant-carrying tube layout, and repeating the analysis using the newcoolant-carrying tube layout.

Aspects of the detailed description presented above are discussed interms of program procedures executed on a computer, a network or acluster of computers. These procedural descriptions and representationsare used by those skilled in the art to most effectively convey thesubstance of their work to others skilled in the art. They may beimplemented in hardware or software, or a combination of the two.

A procedure is here, and generally, conceived to be a sequence of stepsleading to a desired result. These steps are those requiring physicalmanipulations of physical quantities. Usually, though not necessarily,these quantities take the form of electrical or magnetic signals capableof being stored, transferred, combined, compared, and otherwisemanipulated. It proves convenient at times, principally for reasons ofcommon usage, to refer to these signals as bits, values, elements,symbols, characters, terms, numbers, objects, attributes or the like. Itshould be noted, however, that all of these and similar terms are to beassociated with the appropriate physical quantities and are merelyconvenient labels applied to these quantities.

Further, the manipulations performed are often referred to in terms,such as adding or comparing, which are commonly associated with mentaloperations performed by a human operator. No such capability of a humanoperator is necessary in many of the operations described herein whichform part of the present invention; various ones the operations may beautomatic machine operations. Useful machines for performing theoperations of the present invention include general purpose digitalcomputers or similar devices.

Aspects of the invention may be implemented in a high level proceduralor object-oriented programming language to communicate with a computer.However, the inventive aspects can be implemented in assembly or machinelanguage, if desired. In any case, the language may be a compiled orinterpreted language.

Aspects of the invention may be implemented as a mechanism or a computerprogram product comprising a recording medium. Such a mechanism orcomputer program product may include, but is not limited to CD-ROMs,diskettes, tapes, hard drives, computer RAM or ROM and/or theelectronic, magnetic, optical, biological or other similar embodiment ofthe program. Indeed, the mechanism or computer program product mayinclude any solid or fluid transmission medium, magnetic or optical, orthe like, for storing or transmitting signals readable by a machine forcontrolling the operation of a general or special purpose programmablecomputer according to the method of the invention and/or to structureits components in accordance with a system of the invention.

Aspects of the invention may also be implemented in a system. A systemmay comprise a computer that includes a processor and a memory deviceand optionally, a storage device, an output device such as a videodisplay and/or an input device such as a keyboard or computer mouse.Moreover, a system may comprise an interconnected network of computers.Computers may equally be in stand-alone form (such as the traditionaldesktop personal computer) or integrated into another environment (suchas a partially clustered computing environment). The system may bespecially constructed for the required purposes to perform, for example,the method steps of the invention or it may comprise one or more generalpurpose computers as selectively activated or reconfigured by a computerprogram in accordance with the teachings herein stored in thecomputer(s). The procedures presented herein are not inherently relatedto a particular computing environment. The required structure for avariety of these systems will appear from the description given.

Again, the capabilities of one or more aspects of the present inventioncan be implemented in software, firmware, hardware or some combinationthereof.

One or more aspects of the present invention can be included in anarticle of manufacture (e.g., one or more computer program products)having, for instance, computer usable media. The media has therein, forinstance, computer readable program code means or logic (e.g.,instructions, code, commands, etc.) to provide and facilitate thecapabilities of the present invention. The article of manufacture can beincluded as a part of a computer system or sold separately.

Additionally, at least one program storage device readable by a machineembodying at least one program of instructions executable by the machineto perform the capabilities of the present invention can be provided.

The flow diagrams depicted herein are just examples. There may be manyvariations to these diagrams or the steps (or operations) describedtherein without departing from the spirit of the invention. Forinstance, the steps may be performed in a differing order, or steps maybe added, deleted or modified. All of these variations are considered apart of the claimed invention.

Although preferred embodiments have been depicted and described indetail herein, it will be apparent to those skilled in the relevant artthat various modifications, additions, substitutions and the like can bemade without departing from the spirit of the invention, and that theseare therefore considered to be within the scope of the invention asdefined in the following claims. For example, other non-influencingfastener arrangements may be used in lieu of the non-influencingfastener arrangements described above. Moreover, althoughnon-influencing fasteners may be preferable, adhesives may be used inlieu of the non-influencing fasteners described, such as a pressuresensitive adhesive, UV-sensitive adhesive, thermal curing adhesive,epoxy or any other suitable adhesive.

1. A method of configuring a cooling subassembly for an electronicssystem, the method comprising: obtaining a coolant-carrying tube layoutfor interconnecting multiple liquid-cooled cold plates in series-fluidcommunication for cooling multiple heat-generating electronic componentsof an electronics system, the interconnected multiple liquid-cooled coldplates being a cooling subassembly, the multiple heat-generatingelectronic components to be cooled to be plugged in fixed spacedrelation into corresponding sockets on a preconfigured supportingmotherboard, and the coolant-carrying tube layout comprising at leastone rigid coolant-carrying tube; determining for the coolant-carryingtube layout if stress in one or more coolant-carrying tubes thereofexceeds a predetermined acceptable level and determining if loss ofactuation load on a heat-generating electronic component being cooled bythe cooling subassembly exceeds an acceptable loss threshold, and ifeither is true, identifying at least one tube of the coolant-carryingtube layout having high stress, the at least one tube comprising atleast one of a tube portion in torsion and a tube portion in bendingwhen a force is applied to at least one liquid-cooled cold plate of thecooling subassembly, and reconfiguring the at least one tube having highstress to produce a reconfigured coolant-carrying tube layout andrepeating the determining for the reconfigured coolant-carrying tubelayout; performing analysis on the cooling subassembly and the multipleheat-generating electronic components of the electronics system once thecoolant-carrying tube layout experiences tube stress below the definedacceptable level and a loss of actuation load on the multipleheat-generating electronic components below the acceptable lossthreshold, the analysis including employing an available actuation loadon at least part of the cooling subassembly and evaluating electricalconnection loading between at least one heat-generating electroniccomponent being cooled thereby and the supporting motherboard into whichthe at least one heat-generating electronic component electrically plugsin fixed spaced relation; and saving the coolant-carrying tube layout asa final design for interconnecting in series-fluid communication themultiple liquid-cooled cold plates of the cooling subassembly if theelectrical connection loading is above an acceptable minimum level. 2.The method of claim 1, wherein the determining includes performing asimplified analysis on the cooling subassembly to determine for thecoolant-carrying tube layout stress on the at least one coolant-carryingtube interconnecting the multiple liquid-cooled cold plates and loss ofactuation load on at least one heat-generating electronic component ofthe multiple heat-generating electronic components resulting from thecoolant-carrying tube layout, the simplified analysis being performedwithout evaluating electrical connection loading for the multipleheat-generating electronic components into the supporting motherboard.3. The method of claim 2, wherein performing the simplified analysiscomprises performing a simplified three-dimensional numerical analysisof stress and strain on the cooling subassembly, and wherein theperforming analysis on the cooling subassembly and the multipleheat-generating electronic components comprises performing athree-dimensional numerical analysis of stress and strain on the coolingsubassembly and the multiple heat-generating electronic components,including employing the available actuation load on the coolingsubassembly to evaluate electrical connection loading on the multipleheat-generating electronic components and the supporting motherboardinto which the multiple heat-generating electronic componentselectrically connect.
 4. The method of claim 3, wherein performing thesimplified analysis on the cooling subassembly comprises performing asimplified finite element analysis on the cooling subassembly toevaluate stress on the at least one coolant-carrying tubeinterconnecting the multiple liquid-cooled cold plates and loss ofactuation load on the at least one heat-generating electronic componentof the multiple heat-generating electronic components resulting from thecoolant-carrying tube layout.
 5. The method of claim 3, wherein theperforming analysis on the cooling subassembly and the multipleheat-generating electronic components of the electronics systemcomprises performing a finite element analysis on the coolingsubassembly and the multiple heat-generating electronic components,including employing the available actuation load on the coolingsubassembly to evaluate electrical connection loading on the multipleheat-generating electronic components and the supporting motherboardinto which the multiple heat-generating electronic componentselectrically connect.
 6. The method of claim 1, wherein thereconfiguring comprises for the at least one tube having high stress atleast one of introducing at least one horizontal bend into the at leastone tube having high stress, lengthening a tube portion in torsion ofthe at least one tube having high stress, or lengthening a tube portionin bending of the at least one tube having high stress.
 7. The method ofclaim 1, wherein the reconfiguring comprises determining whether alengthened tube portion in torsion for the at least one tube having highstress produces at least one of a lower stress on the at least one tubeor a lower reactionary force on at least one heat-generating electroniccomponent having loss of actuation load beyond the acceptable lossthreshold, and if so, reconfiguring the coolant-carrying tube layout toinclude the lengthened tube portion in torsion and repeating thedetermining for the reconfigured coolant-carrying tube layout,otherwise, determining whether a lengthened tube portion in bending forthe at least one tube having high stress results in at least one of alower stress on the at least one tube or a lower reactionary force onthe at least one heat-generating electronic component having loss ofactuation load beyond the acceptable loss threshold, and if so,reconfiguring the coolant-carrying tube layout to include the lengthenedtube portion in bending, and repeating the determining for thereconfigured coolant-carrying tube layout.
 8. The method of claim 1,wherein the reconfiguring comprises determining whether a distancebetween an outlet and an inlet for at least two series-connectedliquid-cooled cold plates of the multiple liquid-cooled cold plates canbe increased, and if so, redefining a location of at least one of theoutlet and the inlet for the at least two series-connected liquid-cooledcold plates to maximize a path length for a coolant-carrying tubeinterconnecting the outlet and the inlet of the at least twoseries-connected liquid-cooled cold plates while maintaining anypredefined constraints on the location of the outlet and the inlet forthe at least two series-connected liquid-cooled cold plates.
 9. Themethod of claim 1, wherein identifying at least one tube of thecoolant-carrying tube layout having high stress comprises examining tiltof at least one liquid-cooled cold plate when the force is applied to atleast one liquid-cooled cold plate of the cooling subassembly, and forat least one liquid-cooled cold plate having tilt, identifying at leastone tube associated with a highest corner thereof as the at least onetube having high stress.
 10. The method of claim 1, wherein theemploying further comprises performing computational fluid dynamicsanalysis on the cooling subassembly to determine if pressure drop andflow distribution are acceptable through the cooling subassembly priorto selecting the coolant-carrying tube layout as the final design, andwherein the method further comprises reconfiguring the coolant-carryingtube layout if the pressure drop or flow distribution through thecooling subassembly is unacceptable, the reconfiguring including atleast one of reducing a number of bends in the coolant-carrying tubelayout or reducing a length of at least one tube in the coolant-carryingtube layout to create a new coolant-carrying tube layout, and repeatingthe determining, the reconfiguring, the performing, and the saving forthe new coolant-carrying tube layout.
 11. A method of configuring acooling subassembly for an electronics system, the method comprising:obtaining a coolant-carrying tube layout for interconnecting multipleliquid-cooled cold plates in series-fluid communication for coolingmultiple heat-generating electronic components of an electronics system,the interconnected multiple liquid-cooled cold plates being a coolingsubassembly, the multiple heat-generating electronic components to beplugged in fixed spaced relation into corresponding sockets on apreconfigured supporting motherboard, and the coolant-carrying tubelayout comprising at least one rigid coolant-carrying tube; performingsimplified three-dimensional numerical analysis of stress and strain onthe cooling subassembly to determine for the coolant-carrying tubelayout stress on at least one coolant-carrying tube interconnecting atleast two liquid-cooled cold plates of the multiple liquid-cooled coldplates and loss of actuation load on at least one heat-generatingelectronic component of the multiple heat-generating electroniccomponents resulting from the coolant-carrying tube layout, thesimplified three-dimensional numerical analysis of stress and strainbeing performed without evaluating electrical connection loading of themultiple heat-generating electronic components into the supportingmotherboard employing an available actuation load; determining for thecoolant-carrying layout if stress in the at least one coolant-carryingtube exceeds a predefined acceptable level and determining if loss ofactuation load on the at least one heat-generating electronic componentexceeds an acceptable loss threshold, and if either is true,reconfiguring the coolant-carrying tube layout until stress in the atleast one coolant-carrying tube interconnecting the at least twoliquid-cooled cold plates is below the predefined acceptable level andthe loss of actuation load on the at least one heat-generatingelectronic component is below the acceptable loss threshold; performingdetailed three-dimensional numerical analysis of stress and strain onthe cooling subassembly and the multiple heat-generating electroniccomponents of the electronics system once the coolant-carrying tubelayout experiences tube stress below the defined acceptable level andloss of actuation load on the at least one heat-generating electroniccomponent below the acceptable loss threshold, the detailedthree-dimensional numerical analysis of stress and strain includingemploying the available actuation load on the cooling subassembly toevaluate electrical connection loading for the multiple heat-generatingelectronic components into the sockets on the supporting motherboard;and saving the coolant-carrying tube layout as a final design forinterconnecting in series-fluid communication the multiple liquid-cooledcold plates of the cooling subassembly if the electrical connectionloading is acceptable.
 12. The method of claim 11, wherein theperforming simplified three-dimensional numerical analysis of stress andstrain comprises performing a finite element analysis on the coolingsubassembly to evaluate stress on the at least one coolant-carrying tubeinterconnecting the multiple liquid-cooled cold plates and loss ofactuation load on the multiple heat-generating electronic componentsresulting from the coolant-carrying tube layout, and wherein theperforming detailed three-dimensional numerical analysis of stress andstrain comprises performing a finite element analysis on the coolingsubassembly and the multiple heat-generating electronic components,including employing the available actuation load on the coolingsubassembly to evaluate electrical connection loading of the multipleheat-generating electronic components into the supporting motherboard.13. The method of claim 11, wherein the reconfiguring comprises for theat least one tube having high stress, at least one of introducing atleast one horizontal bend into the at least one tube having high stress,lengthening a tube portion in torsion of the at least one tube havinghigh stress, or lengthening a tube portion in bending of the at leastone tube having high stress.
 14. The method of claim 11, wherein thereconfiguring comprises determining whether a lengthened tube portion intorsion for the at least one tube having high stress produces at leastone of a lower stress on the at least one tube or a lower reactionaryforce on the at least one heat-generating electronic component havingloss of actuation load beyond the acceptable loss threshold, and if so,reconfiguring the coolant-carrying tube layout to include the lengthenedtube portion in torsion and repeating the determining for thereconfigured coolant-carrying tube layout, otherwise, determiningwhether a lengthened tube portion in bending for the at least one tubehaving high stress results in at least one of a lower stress on the atleast one tube or a lower reactionary force on the at least oneheat-generating electronic component having loss of actuation loadbeyond the acceptable loss threshold, and if so, reconfiguring thecoolant-carrying tube layout to include the lengthened tube portion inbending, and repeating the determining for the reconfiguredcoolant-carrying tube layout.
 15. The method of claim 11, wherein thereconfiguring comprises determining whether a distance between an outletand an inlet for at least two series-connected liquid-cooled cold platesof the multiple liquid-cooled cold plates can be increased, and if so,redefining a location of at least one of the outlet and the inlet forthe at least two series-connected liquid-cooled cold plates to maximizea path length for a coolant-carrying tube interconnecting the outlet andthe inlet of the at least two series-connected liquid-cooled cold plateswhile maintaining any predefined constraints on the location of theoutlet and the inlet for the at least two series-connected liquid-cooledcold plates.
 16. The method of claim 11, wherein the reconfiguringfurther includes identifying at least one tube of the coolant-carryingtube layout having high stress by examining tilt of at least oneliquid-cooled cold plate during the simplified three-dimensionalnumerical analysis of stress and strain, and for at least oneliquid-cooled cold plate having tilt, identifying the at least one tubeassociated with a highest corner thereof as the at least one tube havinghigh stress.
 17. A method of configuring a cooling subassembly for anelectronics system, the method comprising: ascertaining a configurationof an electronics system, the electronics system comprising apreconfigured supporting motherboard and a plurality of heat-generatingelectronic components to be cooled, the plurality of heat-generatingelectronic components to be cooled being electrically pluggable intocorresponding sockets on the preconfigured motherboard; obtaining anavailable actuation load for plugging the plurality of heat-generatingelectronic components into the corresponding sockets of the motherboardand a required final load of each electrical connection of the pluralityof heat-generating electronic components into the motherboard; andproviding a cooling subassembly employing the ascertained configurationof the electronics system, the cooling subassembly comprising aplurality of liquid-cooled cold plates, the providing including: layingout at least one cooling subassembly, the at least one coolingsubassembly comprising at least two liquid-cooled cold plates of theplurality of liquid-cooled cold plates coupled in series-fluidcommunication for cooling at least two heat-generating electroniccomponents of the plurality of heat-generating electronic components,the at least two liquid-cooled cold plates being coupled in series-fluidcommunication employing at least one rigid tube, wherein the laying outcomprises: selecting a configuration for the at least one rigid tube;and employing the ascertained available actuation load and requiredfinal load on each electrical connection to determine if loss ofactuation load on a heat-generating electronic component to be cooled byone of the at least two liquid-cooled cold plates coupled inseries-fluid communication employing the selected configuration for theat least one rigid tube is below an acceptable threshold, and if so,employing the selected configuration for the at least one rigid tube inthe at least one cooling subassembly, otherwise, reconfiguring the atleast one rigid tube to reduce reactionary force resulting therefrom onat least one liquid-cooled cold plate of the at least two liquid-cooledcold plates coupled in series-fluid communication, said reconfiguringbeing repeated until loss of actuation load on the at least oneliquid-cooled cold plate of the at least two liquid-cooled cold platescoupled in series-fluid communication is below the acceptable threshold.18. The method of claim 17, wherein the employing further comprisesperforming three-dimensional numerical analysis of stress and strain onthe cooling subassembly and the multiple heat-generating electroniccomponents, including employing the available actuation load on thecooling subassembly to evaluate electrical connection loading of themultiple heat-generating electronic components and the supportingmotherboard into which the multiple heat-generating electroniccomponents plug.
 19. The method of claim 17, wherein the reconfiguringcomprises for the at least one rigid tube, introducing at least onehorizontal bend into the at least one rigid tube, lengthening a tubeportion in torsion of the at least one rigid tube, or lengthening a tubeportion in bending of the at least one rigid tube.
 20. The method ofclaim 17, wherein the laying out further comprises performingcomputational fluid dynamics analysis on the cooling subassembly todetermine if pressure drop and flow distribution are acceptable throughthe cooling subassembly prior to finalizing design of the coolingsubassembly, and wherein the method further comprises reconfiguring theat least one rigid tube if pressure drop or flow distribution throughthe cooling subassembly is unacceptable, the reconfiguring including atleast one of reducing a number of bends in the at least one rigid tubeor reducing a length of the at least one rigid tube.